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ABSTRACT 

Reform efforts in mathematics aim to increase 
conceptual understanding, an aim that can be supported through 
concept maps. This study compared the conceptual knowledge of 
function held by college students in reform and traditional calculus 
sections at a large state university. Fourteen students from reform 
sections and 14 from traditional sections served as subjects. A 
primary task was the construction of a concept map of function. Four 
instructors of reform sessions and four from traditional sections 
also completed concept maps. Quantitative analyses of the concept 
maps showed that the core contents from both student groups matched 
poorly with instructors' core concepts. Qualitative analysis of the 
student maps revealed differences between the student groups, with 
the reform group using terminology common in the reform text and 
using fewer algorithmic references than the traditional group. The 
traditional group's maps contained more algorithmic references to 
hand“graphing techniques. Maps of both groups were considerably less 
well-structured than experts' maps and lacked their higher level 
categories. (Contains 9 figures, 10 tables, and 35 references.) 
(Author/SLD) 
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ABSTRACT 



Reform efforts in mathematics aim to increase conceptual understanding. 
Concept maps can serve well as an assessment instrument in this area. This 
study compared the conceptual knowledge of function held by college students 
enrolled in third quarter reform and traditional calculus sections at v. large slate 
research university. Fourteen students from reform sections and fourteen from 
traditional sections served as subjects. A primary task was the construction of a 
concept map of function. Eight PhD's in mathematics, four who taught reform 
classes and four who taught traditional classes, also completed concept maps of 
function. The study compared these expert maps with the student maps. 

Quantitative analysis of the concept maps showed the two student groups' 
core concepts matched up poorly with the experts' core concepts. Neither group 
more closely matched the experts to a significant degree. Qualitative analysis of 
the core concepts and of the maps as a whole revealed differences between the 
student groups. The reform group used terminology common in the reform text 
and had fewer algorithmic references than the traditional group. The traditional 
students' maps contained more algorithmic references to hand-graphing 
techniques. The maps of both student groups were considerable less well 
structured than the experts' maps and lacked the higher-level categories found 
on the expert maps. 
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Introduction 

In the past decade, much change has occurred in the field of mathematics 
education at both the K-12 and the undergraduate levels. Many consider the old 
pattern of teachers prescribing and students transcribing a generally ineffective 
teacning strategy for long-term learning, higher-order thinking, and versatile 
problem-solving. Publications such as Everybody Counts (National Research 
Council, 1989) and the Curriculum and Evaluation Standards (National Council 
of Teachers of Mathematics [NCTM], 1989) have led the way in K-12 curriculum 
reform. They have called for teaching methods that encourage students to 
construct their own mathematical knowledge and have emphasized 
understanding over the rote memorization of algorithms and the acquisition of 
mechanical problem-solving techniques. 

This paper discusses one study of calculus reform in which concept maps 
played an important role as a research tool designed to capture information 
about conceptual understanding. First I present an overview of the research on 
conceptual understanding. Next, I delineate the development of the concept 
map and its variations. A description of the study follows along with the analysis 
of the subjects' concept maps. Finally, I discuss the value of the concept map as 
a research tool in mathematics and make suggestions for further applications 
and research. 




Research on conceptual understanding 

The NCTM Standards (1989) and other reform documents emphasize 
conceptual understanding. Cognitive psychologists seem to agree that the 
internal representation of knowledge resembles webs or networks of ideas that 
are organized and structured (Chi & Koeske, 1983: Hiebert & Carpenter, 1992; 
Hiebert& Lefevre, 1986; Janvier, 1987; Michener, 1978; Nickerson, 1985; 
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Novak, 1977: Pintrich. Marx& Boyle. 1993; Resnick & Ford, 1981: Royer, 

Cisero, & Carlo, 1993). The more connections between facts, ideas, and 
procedures, the better the understanding (Hiebert & Carpenter, 1992: Hiebert & 
Lefevre. 1986: Nickerson, 1985). Individuals whose knowledge is 
interconnected and structured activate large chunks of information when 
performing an activity in the knowledge domain (Fisher & Lipson, 1985; Prawat, 
1989, Royer et al, 1993). A highly integrated knowledge structure hallmarks the 
transition from novice to expert performance (Royer et al, 1993). 

In the current study I focused primarily on conceptual knowledge as 
opposed to procedural knowledge, and especially on understanding and 
meaningful learning as they apply to mathematics. Hiebert and Lefevre (1986) 
define "conceptual knowledge" as knowledge rich in relationships, a network in 
which the linking relationships are as prominent as the discrete pieces of 
information (p. 3). Conceptual knowledge grows in one of two ways: Joining two 
pieces of knowledge already in memory or adding a new piece of information to 
existing structures (Resnick and Ford, 1981 ). As described by Hiebert and 
Lefevre (1986), this construction of new knowledge occurs at two levels. A 
relationship connecting two pieces of information on the same level of 
abstractness constitutes the "primary" level relationship. The "reflective" level 
constructs relationships on a level of abstraction higher than that of the pieces of 
information they connect. This view promotes the idea that relationships in a 
knowledge network may be hierarchical. y 

Following the lead of Greeno (1978), this paper equates "learning with 
understanding" with "meaningful learning." A mathematical concept is 
"understood" when its mental representation becomes part of a network of 
representations (Hiebert & Carpenter, 1992). It follows that "a richness of 
knowledge is needed for deep understanding" (Michener, 1978, p. 374, see also 
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Nickerson, 1985). Conceptual knowledge, defined as a network of relationships 
can be acquired only in a meaningful way. At the opposite end of the continuum 
is rote learning, learning that exhibits few relationships and is closely tied to the 
context in which it is learned. Rote learning of inherently meaningful material 
often takes place when the student does not have sufficient background or time 
to construct a meaningful network of the information (Doyle, 1983; Novak, 1977). 
Development of concept maps 

While conceptual knowledge in mathematics is certainly desirable, how 
does one measure or assess such knowledge? This section briefly discusses 
the history and development of concept maps as tools for looking at conceptual 
knowledge. 

Unlike assessment techniques used in heavily quantitative, primarily 
Dehavioral studies, cognitive techniques often employ both qualitative and 
quantitative measures. A cognitive approach seeks to determine the 
organization and structure of the knowledge base as well as the fluency and 
efficiency with which the knowledge can be used. Concept maps are an 
e.xample of a direct technique used to look at the organization and structure of 
knovv-ledge in this study. While Royer et al. (1993), in their review of techniques 
and procedures for assessing cognitive skills, do not list concept maps, other 
researchers acknowledge it as a "known technique" (Schoenfeld et al, 1993; see 
also Novak & Gowin, 1984; Shavelson, Ruiz-Primo, Lang & Lewin, in press). 

In this study, I found concept maps preferable to other techniques for 
several reasons. First, concept maps maximize subject involvement and 
minimize the researcher’s intrusive role. In drawing and labeling the linking 
lines, the subjects explicitly state the relationships they see. Other direct 
techniques require the researcher to infer what relationship the subject 
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perceived. In the same manner concept maps permit experts to organize their 
knowledge in their own way. This makes comparisons between student and 
experts more accurate and valid. While classifying concept into categories 
produces va‘'d data for some domains, concept maps better depict mathematical 
knowledge and structure, which do not always lend themselves to simple 
categories and subcategories. 

At present, the way one constructs and uses concept maps varies widely. 

In some instances, the subjects draw the maps; in other cases, the researchers 
construct the map from subject protocols. Some researchers require maps to be 
hierarchical; others do not. This diversity arises because concept mapping has 
emerged from two different theories. A short history of concept maps helps to 
explain these variations. 

Novak and Gowin (1984), using the cognitive theory of Ausubel (1968), 
invented the schematic device they called a concept map. The concept map 
represents a set of concept meanings embedded in a framework of propositions 
"Concepts" represent regularities in objects and events and are linked by words 
to form "propositions." Concept maps reflect the theory that conceptual 
knowledge forms a web or network of concepts and the connections between 
them. Novak and Gowin believe concept maps to be explicit, overt 
representations'of the concepts and propositions a person holds, while 
acknowledging the difficulty of judging the degree of correspondence between 
the map and actual internal representation. Certainly subjects cannot draw 
concepts and links they do not possess in memory (unless they have memorized 
a concept map). Consequently, a concept map reveals, at least partially, their 
cognitive structure. 

Because of their own work and their use of Ausubel's learning theory, 
Novak and Gowin (1984) posit that concept maps should be hierarchical, should 
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use linking words on all the conne ng lines, and should be constructed by the 
student. The emphasis on linking words and student construction is significant 
Since the complexity of the knowledge network directly relates to the degree of 
understanding, linking words reveal whether or not students correctly associate 
the concepts--or even if they link them at all. Having students personally 
construct the map provides more accurate data than having a researcher draw 
the maps using a student interview protocol (assuming the students are old 
enough to understand concept map construction). 

Another non-hierarchical theory of cognitive structures developed about 
the samt= time as Ausubel's. Using word-association techniques based upon the 
theory of James Deese (1965), researchers represented concepts as nodes in a 
network but did not label the connecting lines. They used indirect methods 
(word associations, similarity judgments, and tree building) for eliciting 
representations. This school of thought evolved into semantic network theory 
where nodes are connected by directional and labeled lines to produce 
propositions. It proposed to account for the kinds of statements people were 
willing to make about a topic, as well as the relative speed with which they made 
them as measured in reaction time experiments (Resnick & Ford, 1981). 
However, the theory did not live up to its early promise (Johnson-Laird, 1983), 
since it failed to predict time differences within categories. 

When one puts labels oh the lines to produce propositions, semantic 
networks resemble Novak and Gowin's concept maps-with the exception that 
they do not have to be hierarchical (Shavelson et al., in press). The concept 
map theory, however, makes no claim about the rate of access to information. 
Studies by Fisher (Fisher, 1990; Fisher & Lipson, 1985) make it clear that some 
researchers have moved from an exclusively semantic network terminology to an 
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incorporation of Novak and Gowin's concept map terminology with references to 
Ausubel's theory. 

Theoretical distinctions do exist for other researchers, particularly in 
regard to whether or not the knowledge representation must be hierarchical. 
Ausubel’s (1968) theory of meaningful learning describes a hierarchical structure 
of conceptual knowledge, and those studies emanating from the Novak and 
Gowin tradition adhere strictly to the hierarchical requirement with the most 
general topic at the top of the map. White and Gunstone (1992) argue that 
whether or not a concept maps should be hierarchical depends on the structure 
of the subject matter. Researchers from the semantic network tradition tend 
toward a map with a general concept in the center and with links coming out 
much like the spokes of a wheel. Harnisch call them "spider maps" (Harnisch et 
al, 1994). In many cases the spider configuration can be redrawn to show a 
hierarchical relationship. However, because the question of hierarchy is not fully 
resolved, I did not assume or demand hierarchical structures in this study. 

Seven recent studies used concept maps to assess cognitive structure or 
conceptual understanding (Beyerbach, 1986; Coleman, 1993; Laturno, 1994; 
Markham, Mintzes, & Jones, 1994; Park, 1993; Rogers, in press; Wallace & 
Mintzes 1990). They adhere to the Novak and Gowin tradition for constructing 
concept maps and thus require hierarchical maps. Two studies in particular 
(Markham et al. (1994); Wallace and Mintzes (1990)) address the use of concept 
maps as an assessment activity in science while two others (Laturno (1994) & 
Park (1993)) use concept maps in studies pertaining to mathematics. 

Wallace and Mintzes (1990) looked at college students (n= 91 ) enrolled m 
an elementary science methods course. They examined the validity of concept 
mapping as an evaluation approach. Their conclusions support the concurrent 
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validity of concept maps as vehicles for documenting and exploring conceptual 
change in biology. 

In a follow-up study, Markham et al. (199*^) looked for more evidence of 
concurrent validity. Their subjects were advanced college biology majors (n = 

25) and beginning nonmajors (n = 25). Again the authors conduderl that their 
results showed further evidence for the concurrent validity of concept mapping 
as a research and evaluation tool in science education. 

Conspicuous by their absence in the large number of studies using 
concept maps are studies in mathematics. Park (1993), in her evaluation of a 
new computer laboratory calculus course at the University of Illinois, does 
provide one example. She gave the students in both a traditional section and a 
computer section of the course a list of concepts to use in making hierarchical 
concept maps. Park scored the maps numerically and used a software program 
to compare the students' maps with a teacher's map. Park's concept map scores 
for the two calculus groups lists no significant differences in any scoring 
category or any total although the reform group's scores were generally higher 
and showed stronger congruence with the teacher's concept map. She also 
found a strong correlation between the concept maps scores and the post- 
achievement test scores (0.82). Park's qualitative analysis of the maps 
determined differences in the concepts each group added to its maps. The 
computer group gave many more visually-oriented terms, while the traditional 
group gave more application and technique-related terms. The linking words 
were equally low-level on all the maps. 

Laturno's (1994) study sought to determine if concept maps corresponded 
with clinical interviews in determining concept connectivity and to see if concept 
maps could predict academic achievement. Her subjects (n=1 1 8) were 
community college students in self-paced, remedial mathematics courses. 
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Laturno followed the tradition of Novak and Gowin and instructed the students 
on hierarchical maps. She used a scoring scheme that gave points for 
additional concepts, relationships, levels of hierarchy, examples, and cross- 
links, Laturno's results indicate that student-generated concept maps show 
indications of validity as a research tool. The concept maps gave results 
comparable to both interviews concerning student knowledge of relationships 
between concepts and to the academic progression of students through the 
course. 

Purpose of study 

The work reported here is part of a larger study that included evaluation 
of concept maps as assessment tools. The study also compared students' from 
reform and traditional calculus classes conception of function. An example- 
nonexample questionnaire looked at students' concept image of function. This 
paper primarily documents the place of concept maps in the study, Williams (in 
press) reports other results and analyses, 

A state university of over twenty thousand students served as the setting 
for this study. It is a top-tier research university and, as such, has high 
admission standards for its undergraduates. For the school year of this study, 
(1993-1994), the university had two, three-quarter sequences of first-year 
calculus, the traditional and the reform. These sequences differed in at least 
four observable ways; 1 ) the textbook; 2) the technology required; 3) the types 
of problems assigned; and 4) the use of written group projects. 

Two groups of subjects participated in the study. One group consisted of 
students enrolled in the reform and traditional calculus classes. The student 
subjects for the study were 28 volunteers enrolled in the third quarter of calculus 
at the large state university. Fourteen came from two reform sections; fourteen 
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came from a single traditional section. Each group (traditional and reform) had 
seven women and seven men. Each student completed two tasks; 1 ) drawing a 
concept map for the topic of function and 2) completing an example-honexample 
questionnaire about function in an audio-taped interview. 

Each student attended a session in which I presented instruction on 
concept maps. Some sessions involved a small group of students, while other 
times i met with a single student, depending on the students' availability. I 
explained concept maps in each session, showing them examples and stressing 
the importance of linking words on the lines. The examples included hierarchical 
maps, web or spider maps, and non-hierarchical maps. i told the students they 
could draw their maps however they wished. After the brief instruction, the 
students each drew practice maps using some concepts I gave them about 
fractions. When I was sure they co ;‘ j draw concept maps, I asked each student 
to bi linstorm and come up with a list of terms related to function. After several 
minutes, I instructed them to draw a concept map for function using the terms 
they had generated as well as any others that .might occur to them. No student 
interaction occurred in any group sessions. The subjects were able to work on 
the maps as long as they desired. All the students took less than an hour to 
complete the maps. 

The other group of subjects included eight professors (PhD's in 
mathematics) at two different universities. Four taught at the large state 
research university the student subjects attended. The other four taught at a 
small, private, west-coast university. Two professors at each school taught the 
reform text and used graphing technology the year of the study. The other two 
professors at each school have taught calculus at this level, but only from 
traditional texts. 
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I gave the experts two concept map tasks. The "unrestricted" task was to 
draw a concept map for function from their perspective as a mathematician. The 
"restricted" task was to draw a concept map of function that represented what 
they would expect students completing the first-year calculus sequence to know. 

I met with each expert and explained concept maps to them in the same manner 
I did the students, showing them the same examples. I asked them to 
brainstorm to get their starting terms. I also gave the experts a 45-minute time 
limit to complete each map, as 1 felt this would help them focus on what could 
otherwise be considered an unlimited task. I did not stay with the experts while 
they worked on the tasks. Rather, I let them complete and return the maps at 
their convenience. They did not know what the second task was until after they 
had completed the first. Half of the experts did the restricted task first (one 
professor from the reform, one professor from the traditional at each school), 
while the other half completed the unrestricted task first. 

Results and Analysis Using Concept Maps 

Much of the analysis in this study focused on differences between groups 
of subjects, students as well as experts. Did concept maps reveal differences 
about the concept of function held by students in reform sections of calculus and 
in traditional sections of calculus? (For brevity and clarity, I will call these 
groups of students the "reform students" and the "traditional students.") I 
particularly looked for differences that might be attributable to different curricula 
that are clearly shown by concept maps. I also compared the concept maps of 
the two groups of experts. I looked to see if their knowledge about function, as 
exhibited in the-ir concept maps, transcended any curricular differences or if they, 
too, may have been influenced by the curriculum they used, 
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Most researchers using concept maps have devised a scoring scheme to 
assign a numerical value to each map. The categories used for scoring often 
include valid propositions, levels of hierarchy, and cross-links. T>^ey 
occasionally include examples. After studying this set of data, I concluded it did 
not lend itself to a valid method of scoring. Several reasons support this 
conclusion. 

First, the task I assigned— to create a concept map about function—was 
purposely unstructured for both the students and experts. I did not ask them to 
make a hierarchical map, nor did I tell them to organize related concepts 
together. I did not give them any concepts as examples, nor any hints as to 
what type of concepts I might expect. None of the students had any prior 
experience with concept mans, and only one expert said he had done concept 
maps. Consequently, while the maps reflect only the students' or experts' work 
and only their personal notions about functions, the maps proved to be widely 
divergent and complex and did not lend themselves to a numerical scoring 
scheme. For instance, the experts generated 197 different concepts on their 
unrestricted maps of function and 1 33 different concepts on their restricted 
maps. The students' maps produced just over 300 different terms. Figure 1 and 
Figure 2 show reduced copies of experts' maps. While the concepts and links 
are difficult to read, the overall complexity of the maps' structure is clearly 
visible. Figure 3, Figure 4, and Figure 5 are reduced copies of the maps for 
three students. These maps once again illustrate the diversity and complexity of 
some of the maps. 

Some might argue that the diversity in the students' maps resulted 
because they did not understand the construction of concept maps. However, 
each student constructed a practice map, and I checked each one personally to 
be sure the student understood the basic procedure. All the student maps show 
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that the students understood that concepts about functions went in the ovals 
while linking words denoting relationships went on the lines. Lacking any 
evidence of misunderstanding, one logically concludes that the wide diversity of 
the maps derives mainly from students' different conceptions about function, 
rather than from any difficulties they had with concept mapping. 

Thus for this particular set of data, the numerical scoring schemes 
typically used do not appear to be valid. (For further discussion, see Williams, 
in press.) By developing lists of concepts most frequently used by the various 
groups of subjects, however, one can use the maps to make some valid 
quantitative comparisons. 

Since the concept map task in this study was unstructured, many 
concepts emerged. In order to compare the students' concepts with those of the 
experts and to compare those of the student and expert subgroups with each 
other. I generated several "core" lists of concepts. Core lists consist of the 
concepts most frequently found on the concept maps of a given subject group. I 
detail the process below. 

As mentioned, the experts each created two concept maps. For the 
restricted task, they constructed a map of what they felt students completing 
first-year calculus should know about function. For the unrestricted task, their 
map was to depitt their view of function as a mathematician. I categorized the 
experts into three groups--reform experts, traditions! experts, and combined 
experts. Since the three expert groups completed two concept maps, they 
generated six lists of concepts. I generated a core concept list under each of 
these conditions. The following sequence shows how I created three core lists 
for the restricted concept mapping tasi 

To form a core list, 1 first compiled a list of the concepts used in the 
individual maps by each expert. Next I combined the reform experts' concepts to 
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Reform Experts Traditional Experts 




Figure 6: Formation of Core Lists 



form a list--"reform experts' complete list" and the traditional experts' concepts to 
form "traditional experts' complete list" (see Figure 6). Putting these two lists 
together yielded the "combined experts' complete list"--a list of all terms used by 
the eight experts. If two or more reform experts (50%) used a concept on the 
reform experts' complete list, I put it on the "reform experts' core list." I used the 
same criterion for compiling the "traditional experts' core list." To form the 
"combined experts' core list," I chose all concepts that appeared on three of the 
eight experts' (38%) lists. (A natural break occurred at the 3-expert criterion with 
25 out of 197 concepts used by a least three experts, 21 concepts used by 
exactly two, and 151 used by only one.) Using the same procedure and same 
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criteria, I generated the three core lists for the unrestricted task. The six core 
lists, along with the number of experts naming each concept, appear in Table 1 . 

The core lists for the restricted tasks were much longer than those for the 
unrestricted tasks, indicating the experts were in better agreement about the 
function concept as taught in first-year calculus than about function in general. 
The reform experts' core list for the restricted task contains 25 concepts, while 
for the unrestricted task the list has only 14. The traditional experts' core list for 
the restricted task has 23 concepts, while for the unrestricted task the list 
contains 8. Using the 3-out-of-8 criterion, the combined perts' core list on the 
restricted task includes 25 terms, while its corresponding core list on the 
unrestricted has 10. These core lists served as benchmarks to which I 
compared the students' concepts. 

I created core lists for the two groups of students in the same manner I 
had the expert core lists. I used a criterion of 7 out of 14 (50%) for the individual 
groups, reform and traditional, and a criterion of 10 out of 28 (36%) for the 
combined group. This process yielded three core lists: "reform students' core 
list", "traditional students’ core list", and "combined student's core list" (see 
Table 2). They include 9, 8, and 10 concepts, respectively. 

Analysis Using the Cora Lists 

Having compiled these lists, I was able to compare the various groups m 
several ways. The quantitative analysis primarily looked for agreement between 
the stud';:nt groups' lists and the expert groups' lists and forms the first part of 
this analysis. The next part of the analysis details the qualitative differences 
between the two student groups and then between the students and experts 
The analysis section concludes with a qualitative comparison of the concept lists 
of the two expert groups. 
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Table 2 

Student Groups' Core Concept Lists and 
Number of Students Choosing Each Concept 

Reform Students' Traditional Students' Combined Students' 

Core List Core List Core List 



function 


14 


function 


14 


function 


28 


derivative 


13 


variables 


11 


derivative 


22 


graph 


12 


derivative 


9 


variables 


21 


slope 


11 


graph 


9 


graph 


20 


variables 


10 


integral 


9 


equation/ 

symbolically 


15 


equation/ 

symbolically 


9 


limit 


7 


integral/ 

integration 


15 


input 


7 


max/min 


7 


max/min 


13 


integral 


7 


slope 


7 


slope 


13 


line/linear 


7 






line/linear 


10 










polynomial 


10 



I compared the students' core concept lists to the experts' core concept 
lists in two ways. First, I took each of the three student core lists and computed 
a ratio to reflect the number of students' concepts matching the experts' 
concepts to the total number of concepts on the expert list (see Table 3). For 
example, of the nine concepts on the reform students' core list, five of them are 
on the reform experts' core list (restricted task) that has 25 concepts, giving a 
ratio of 5 to 25 or 20%. (On Figure 7, this is the ratio of c to b.) Thus the 
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on student list on expert list 

Figure 7: General Relationship of Lists and Number Concepts Used to Compute 
Comparison Percentages 

percentages in Table 3 represent the portion of the experts' lists which the 
students also used. 

This comparison yielded several observations. First, in general, the 
students' core lists do not match the experts' lists to a high degree. The 
correspondence is less than 40% for all comparisons, with most in the 20% 
range. Each of the three groups of students' core concepts matched the 
traditional experts' lists a little better than the other expert groups on both the 
restricted and unrestricted tasks. This may indicate the students know and use 
a higher percentage of concepts similar to those of the traditional experts. 
Although several percentages for the unrestricted task are higher than their 
corresponding percentages on the restricted task, one cannot conclude that the 
students' maps more closely resemble the experts' maps on the unrestricted 
task. One must take into account that the number of concepts on the core lists 
for the unrestricted task are much smaller. Consequently, a single concept can 
represent as much as 12%. Yet another comparison more accurately 
determines which expert mapping task that the students' maps most closely 
resemble. 
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I computed a second ratio: the number of students' concepts matching 
the experts' concepts to the total number of concepts on the student list (see 
Table 4). Again, using the same example, 5 of the 9 concepts on the reform 
students' core list match those on the reform experts' core list (restricted task), 
but this time the comparison is made to the nine-concept student list for a 5 to 9 
or 56% ratio. (In Figure 7 this is the ratio c to a.) Thus Table 4 values show the 
percentage of student groups' core concepts that were also on the various 
experts' lists. 

These figures show that a greater proportion of all the student lists match 
the experts' restricted task lists more closely than they match the experts' 
unrestricted task lists. The range for the restricted task comparisons is 43% to 
67%, while for the unrestricted task, the range of match is only 21% to 38%. 

This result was expected, since the restricted task asked for what experts 
thought first-year calculus students should know. 

Comparing core lists in this manner provided a quantitative measure by 
which to compare the concepts of the various groups. From a broad 
perspective, one can see that one-half to two-thirds of the students' core 
concepts are on the experts' core lists (restricted task) and yet they account for 
no more than 38% of any expert list, indicating disparity between the students 
and experts when it comes to concepts relating to function. Using this 
comparison, one also finds little difference between reform and traditional 
students' core concepts in their relation to the experts' core 'oncepts. 

Since creating composite lists such as the core lists may have masked 
some differences, I also looked at how the individual students' concepts 
compared to the experts' concepts on their core lists and their complete lists. I 
limited the comparison to the experts' restricted task, since it resembles the 
students' maps most closely. 
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In the first comparison, I determined for each student the number of 
concepts that matched each expert core and complete list. Using t-tests I found 
no significant differences between the two student groups in the number of 
concepts individuals used that matched concepts on the expert core concept 
lists. 

In a manner similar to that described above, 1 found the percentage of 
each student's concepts that matched those on the experts' core lists (in Figure 
7 this is the ratio c to b) as well as the number matching the experts' complete 
list of concepts (see Table 5). Looking at Table 5, one can see that the student 
lists corresponded better to the experts' core lists than to the experts' complete 
lists. Table 6 summarizes the comparisons using these individual match 
percentages. The mean percentage of match to the experts' core concepts 
ranged between 23% and 31 %. The mean percentage of match to the experts' 
complete list of concepts ranged between 7.6% and 13.3%. Both the reform and 
traditional student groups matched the traditional experts' core and complete 
lists slightly better than they did the corresponding reform experts' lists. This 
finding is consistent with student core list comparisons reported above. 

Another way in which 1 compared the student groups was to look at whai 
percentage of each individual student's concepts was also on any of the experts' 
core or complete* lists (in Figure 7 this is the ratio c to a). Table 7 shows the 
percentages for each student as well as the number of concepts each student 
map contained. In this comparison, the percentages for the match to the 
experts' complete lists are higher than those for the core lists. Since the 
combined experts' complete list consists of all terms used by the experts. 

Column H of Table 7 is a good indication of how meaningful and relevant the 
students' concepts are if one considers all expert concepts to be meaningful and 
relevant. 3 7 
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Table 5 



Individual Student's Percentage Match to Expert Lusts 
Restncted Task 



A 1 B 1 


c 


D ■ E 


F G 


No of 
ConccDts 


25 


23 


25 


97 


75 1 


133 


R1 


24% I 26% 


20%) 


11%) 


11%) 


10%) 


R2 


24% I 26% 


24%) 


8%) 


11%) 


7%) 


R3 


24% I 26% 


20%) 


9% 


12%) 


8%) 


R4 


24% 


30% 


24%) 


6%) 


13%) 


3% 


R5 


16% 


26% 


24%) 


9% 


12%) 


8%) 


R6 


32% 


30% 


32%) 


9%) 


11% 


7% 


R7 


32% 


39% 


32%) 


12%) 


17%) 


13%) 


R8 


20% 


22% 


16% 


5% 


8%) 


5% 


R9 


12% 


17% ' 


12% 


6%) 


8%) 


6% 


R10 


48% 


57% 


40%) 


13%) 


21% 


14%) 


R11 


8% 


13% 


8%) 


3"^ 


5% 


3®b 


R12 


20% 


26% 


20% 




11%) 


7%) 


R13 


16% 


22% 


16%) 


7% 


-»1%) 


7%) 


R14 


28% 


30% 


28%) 


8%) 


13%) 


8%) 


T1 


24% 


30% 


24%) 


1 % 


1 11% 


6% 


T2 


28% 


35% 


i;; 

1^ 

L 


OOA 
u . , 


17% 


10%) 


T3 


28% 


35% 


28%) 


9%) 


17%) 


11%) 


T4 


28% 


22% 


28% 


9% 


11%) 


7% 


T5 


28% 


43%) 


28% 


7% 


20% 


11%) 


T6 


8% 


9% 


8%) 


3% 


7% 


4%) 


T7 


36% 


39%) 1 40%) 


12%) 


16%) 


11% 


T8 


40% 


43%) ‘ 36%) 


11%) 


20% 


12%) 


T9 


20% 


26% 1 20% 


5% 


9% 


5% 


T10 


16% 


13%) 


16% 


4 %) 


1 7% 


5% 


T11 


48% 


57%) 


52% 


14%) 


21% 


13%) 


T12 


20% 


30%) 


24%) 


6%) 


13%) 


8%) 


T13 


20% 


26%) 


20%) 


6% 


9%) 


6% 


. TU 


16% 


22% 


16% 


4% 


8%) 


5%) 



Column Codes; 

A: Student Designation 
B: Match to Reform Experts' Core List 
C: Match to Traditional Experts' Core List 
D: Match to Combined Experts' Core List 
E. Match to Reform Experts' Complete List 
F: Match to Traditional Experts' Complete List 
G Match to Combined Experts' Complete List 
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Table 6 

Comparison of Percent Match of Individual Student's Concepts to 
Expert Usts-Restricted Task 



Comparison to Reform Experts' Core Concepts 





Reform Students 


Traditional Students 


Mean 


23.4% 


25.7% 


Variance 


0.9% 


1.1% 


Range 


8%-48% 


8%-48% 



Com 


parison to Traditional Experts' Core Concepts 




Reform Students 


Traditional Students 


Mean 


28.0% 


30.7% 


Variance 


1 .0% 


1 .6% 


Range 


13%-57% 


9%-57% 



Com 


parison to Combined Experts' Core Concepts 




Reform Students 


Traditional Students 


Mean 


22.6% 


26.3% 


Variance 


0.7% 


1 .2% 


Range 


8%-40% 


8%-52% 



Comparison to Reform Experts' Complete List 





Reform Students 


Traditional Students 


Mean 


8.3% 


7.7% 


Variance 


0.1% 


0.1% 


Range 


3%-13% 


3%-14% 



Comparison to Traditional Experts' Complete List 





Reform Students 


Traditional Students 


Mean ‘ 


11.7% 


13.3% 


Variance 


0.2% 


0.37o 


Range 


55-2170 


77o-21% 



Comparison to Combined Experts' Complete List 





Reform Students 


Traditional Students 


Mean 


7.6% 


7.97o 


Variance 


0.1 7o 


0.1% 


Range 


37o-14% 


4%-137o 
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Table 7 



Percentage of Individual Student's Concepts 
That Matched Expert Lists (Ri^sthcted Task) 



A 1 


B 


C 


D 


E 


F 


G 


H 


R1 


22 


27% 


21% 


23®/o 


50®/o 


36®/o 


59®/o 


R2 


31 


19% 


19®/o 


19®/o 


26®/o 


26®/o 


29®/o 


R3 


21 


29% 


29®/o 


24®/o 


43®/o 


43®/o 


52®/o 


R4 


29 


21% 


24®/o 


21% 


21®/o 


34®/o 


14®/o 


R5 


26 


15®/ 


23®/o 


23®/o 


35®/o 


35®/o 


42®/o 


R6 


21 


38®/o 


33®/o 


38®/o 


43®/o 


38®/o 


43®/o 


R7 


27 


30®/o 


33% 


30®/o 


44®/o 


48®/o 


63®/o 


R8 


17 


29®/o 


29®/o 


24®/o 


•29®/o 


35®/o 


41®/o 


R9 


23 


13®/o 


17®/o 


13®/o 


26®/o 


26®/o 


35®/o. 


R10 


32 


38®/o 


41®/o 


31®/o 


41®/o 


50®/o 


59®/o 


R11 


29 


7% 


10®/o 


7% 


10®/o 


14®/o 


14®/o 


R12 


22 


23®/o 


21% 


23®/o 


32®/o 


36% 


41®/o 


R13 


22 


18®/o 


23®/o 


18®/o 


32®/o 


36% 


41®/o 


R14 


36 


19®/o 


19®/o 


19®/o 


22% 


28% 


31®/o 


T1 


31 


19®/o 


23®/o 


19®/o 


23®/o 


26% 


26®/o 


T2 


44 


16®/o 


18®/o 


16% 


18®/o 


30®/o 


30®/o 


T3 


41 


17®/o 


20®/o 


17®/o 


22% 


32®/o 


34®/o 


T4 


20 


35®/o 


25®/o 


35% 


45®/o 


40®/o 


45®/o 


T5 


28 


25®/o 


36®/o 


25% 


25®/o 


54®/o 


54®/o 


T6 


19 


11®/o 


11®/o 


11®/o 


16®/o 


26®/o 


26®/o 


T7 


35 


26®/o 


26®/o 


29®/o 


34®/o 


34% 


40®/o 


T8 


29 


34®/o 


34®/o 


31% 


38®/o 


52% 


55®/o 


T9 


17 


29®/o 


35®/o 


29®/o 


29®/o 


41®/o 


41®/o 


T10 


12 


33®/o 


25% 


33% 


33®/o 


42®/o 


50®/o 


T11 


36 


33®/o 


36®/o 


36®/o 


39®/o 


44®/o 


47®/o 


T12 


45 


11®/o 


16®^ 


13®/o 


13®/o 


22% 


22% 


T13 


15 


33®/o 


40®/o 


33% 


40% 


47®/o 


53®/o 


T14 


24 


17®/o 


21®/o 


17®/o 


17®/o 


25®/o 


25®/o 



Column Codes: 

A: Student Designation 

B: Number of Concepts Generated by Student 

C: % Student's Concepts on Reform Experts' Core List 

D: % Student's Concepts on Traditional Experts' Core List 

E: % Student's Concepts on Combined Experts' Core List 

F; % Student's Concepts on Reform Experts' Complete List 
G: % Student's Concepts on Traditional Experts' Complete List 

H: % Student's Concepts on Combined Experts' Complete List 



Comparison of the two student group's number of concepts generated 
using a t-test revealed no significant difference. Although the traditional 
students generated more concepts on average (28.3 concepts was the mean for 
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the traditional students, 25.6 the mean for the reform), this difference was not 
significant at the 0.05 level. Table 8 summarizes the data about the percentage 
of the students' concepts that matched the experts'. In looking at the proportion 
of the students' concepts that were on the experts' complete lists, each group 
matched its corresponding expert group to a higher degree than its peers. 

This analysis generally agrees with the insights gleaned from the student 
core list comparisons. It is useful, however, because comparing the students' 
individual percentages of concepts that are on the expert list approximates 
assigning each map a numerical score for number of relevant and meaningful 
concepts. 

The data presented in Tables 7 and 8 also provides further evidence of 
how difficult it would be to score the students' concept maps numerically. If one 
assumed that a concept mentioned by any expert was meaningful, the majority of 
the students' concepts (which did not match those of experts) would have to be 
judged meaningless or irrelevant. Can a proposition be considered meaningful if 
it contains one or more concepts that are not meaningful? How would one 
assign such "partial credit?" How would one judge levels of hierarchy if some of 
the propositions in the chain were invalid? Once again, scoring this data 
presents serious problems that might be solved arbitrarily but never 
satisfactorily. 

In summary, one can draw several important conclusions from 
quantitative analyses of the concept lists: 1 ) In general, the core concepts of the 
reform students' lists matched the experts' lists to the same degree that 
traditional students' lists did. Student lists from both groups matched traditional 
experts slightly better than reform experts. 2) Both student groups' core 
concepts matched a relatively low percentage of the experts' lists. 3) When 
considering the proportion of tlie individual student's concepts that matched 
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Table 8 

Comparison of Portion of Individual Student's Concepts That 
Matched Expert Lists-Restiicted Task 



Comparison of Number of Concepts Generated 





Reform Students 


Traditional Students 


t-Statistic 


Mean 


25.6 


28.3 


-0.838 


Variance 


28.1 


118.7 




Range 


17-36 


12-45 




Comparison to Reform 


Experts' Core Concepts 




Reform Students 


Traditional Students 


t-Statistic 


Mean 


23.3% 


24.3% 


-0.295 


Variance 


0.8% 


0.8% 




Range 


7%-38% 


11%-35% 




Comparison to Traditional Experts' Core Concepts 




Reform Students 


Traditional Students 


t-Statistic 


Mean 


25.4% 


26.0% 


-0.186 


Variance 


0.6% 


0.8% 




Range 


10%-33% 


ir/o-40% 




Comparison to Combiner 


Experts' Core Concepts 




Reform Students 


Traditional Students 


t-Statistic 


Mean 


22.3% 


24.6% 


-0.738 


Variance 


0.5% 


0.8% 




Range 


7%-38% 


11%-36% 





Comparison to Reform Experts' Complete List 





Reform Students 


Traditional Students 


t-Statistic 


Mean 


33.2% 


28.0% 


1.253 


Variance 


1 .2% 


1.1% 




Range 


10%-44% 


13%-45% 





Comparison to Traditional Experts' Complete List 





Reform Students 


Traditional Students 


t-Statistic 


Mean 


33.7% 


36.7% 


-0.539 


Variance 


0.9% 


1.1% 




Range 


14%-50% 


22%-54% 





Comparison to Combined Experts' Complete List 





Reform Students 


Traditional Students 


t-Statistic 


Mean 


40.3% 


39.2% 


0.212 


Variance 


2.3% 


1 .4% 




Range 


14%-79% 


22%-55% 
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those on experts' list, the reform students and traditional students performed 
about the same, 4) The individual students' proportion of match to the experts' 
complete list of concepts is an indicator of the number of meaningful and 
relevant concepts the students used. As such it provides a way to numerically 

compare a portion of the students' concept maps. 

Qualitative Analysis of Core Concepts 

While quantitative comparisons provide information about how many or 
what proportion of concepts are the same or different for various groups, 
qualitative analysis looks for substantive similarities and differences, I 
qualitatively compared the experts' core concept lists with the students' core lists 
with interesting results, A similar analysis of the concepts with respect to the 
students' coro lists did highlight differences in the two student groups. The final 
comparison in this section looks for differences between the reform and 
traditional experts. 

The quantitative analysis of the concept lists revealed low 
correspondence of students' concepts with experts' concepts. Comparing 
students' concepts with experts' concepts on a qualitative basis sheds some light 
on why this disparity exists. Since earlier analysis showed little difference 
between the reform and traditional students' agreement with the experts, I used 
the combined experts' core concepts and the combined students' core concepts 
for the qualitative comparison. As has been stated, the experts produced 25 
core concepts, the students only 10 (see Table 9), Six of the 10 student 
concepts are on the experts' list. Of particular interest, however, are certain 
items on the experts' list that seldom appear on any student's map. 

One group of concepts from the experts' list-domain, range, and 
correspondence-relates to the definition of a function. None of these terms 
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Table 9 

Combined Core Concept List for Experts and Students 



Combined Experts' Core Combined Students' Core 

Concepts Concepts . 



function 




function 


domain 


derivative 


graph/graphical 


variables 


range 


equation/symbolically 


composition 


integral/integration 


exponential 


max/min 


Inverse 


slope 


polynomial 


line/linear 


trigonometric 


polynomial 


continuous 






derivative 


Concepts given in descending 
order by frequency of use 


logarithmic 


symbolic/equation 


1-1 


classes/common/types/familiar 


differentiation/differentiate 


integral/antiderivative 


adding/dividing/arithmetic 

operations 


correspondence 


integrate/integration 


limit 


oroperties 


rate of change/change 


rational 


real world 



appears on the students' combined core list (nor on t’le other two student core 
lists). Seven of the eight experts (88%) listed domain and range, indicating that 
these concepts form an integral part of their view of function. This does not hold 
true for the students. 
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Another group of concepts on the experts' combined list fits together 
nicely. Exponential, polynomial, trigonometric, logarithmic, and rational are 
all classes or common types of functions. Only polynomial, the simplest and 
most pervasive function in high school algebra, made the students' combined 
list, although undoubtedly the students have encountered all these different 
types of functions. Trigonometric and rational appeared only on student maps 
from the traditional classes (and rational was sometimes used to mean rational 
number, not rational function). Only two students (7%) seemed to set up a class 
or type grouping of functions on their concept maps. One student had 
polynomial and rational as her two branches, another had linear, exponential, 
and quadratic as his. Five of the experts (63%) had such a grouping on their 
maps. 

A third group emerging from the experts' core list involves properties of 
functions, 1-1, continuous, differentiable, and having an inverse. Thirty-eight 
percent of the experts had such a group. Once again, none of these concepts 
figured on the students' combined list. Only one student from the traditional 
group used 1-1, continuous, and inverse, while four others listed continuous 
alone. One student listed inverse. No student used differentiable. 

The experts' list yielded a fourth cluster of concepts: operations one 
performs on functions. These include composition, differentiation, 
integration, and combining with arithmetic operations. Determining the 
inverse of a function might also be considered a part of this group. Only one 
student used differentiation or differentiate; four used integrate or 
integration: two mentioned composite; and one said functions could be added, 
subtracted, multiplied, or divided. No student concept map showed any 
indication of an operations group, while four of the eight (50%) expert maps 
showed a strong operations grouping. To summarize this comparison of the 
o 
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TablelO 

Concepts Used Primarily by One Student Group 

Concepts Used by Reform 
Students 



*3 students from this group are the only 
students who used this concept. 

of the 9 students using the 
concept were from this group. 



behavior (3)* 


contour lines (2) 


evaluated (3) 


global (2) 


implicit (2) 


increasing/decreasing (3) 


level sets (2) 


local linearity (3) 


numerically (3) 


real life (3) 


slope of tangent (3) 


symbolically (4) 


table (5) 


understanding (2) 


input/output (7 of 9)** 


geometrically (2) 


algebraically (1 ) 



Concepts Used by Traditional 
Students 



absolute (4; 


asymptotes (4) 


defined (6) 


rational (5) 


real numbers (3) 


trigonometric (3) 


undefined (4) 


volume (4) 


domain (6 of 8) 


numbers (6 of 8) 


shape (3 of 4) 


f(x) (3 of 4) 


velocity (4 of 5) 


range (5 of 7) 



ERIC 



experts and students, one can say that the experts' core lists showed higher- 
level groupings that are not present in the students' concepts. 

The three student core lists are very similar (see Table 2). The 
differences between the two student groups pertain to concepts that did not 
make the core lists. Table 1 0 shows the concepts used only by one group of 
students or primarily by one group. The resulting sets of concepts illustrate the 
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distinctions in the two teaching methods. For example, asymptotes, defined, 
rational, real numbers, undefined, and domain-concepts used only by 
traditional students-all relate to graphing by hand. To graph a rational function, 
one must determine which real numbers must be excluded from the domain. At 
the values for which the denominator is zero, the function is undefined, and an 
asymptote may exist. Velocity and volume, again terms used by traditional 
students, represent typical applications used in traditional texts for derivatives 
and integrals. On the other hand, the concepts from the students in the reform 
group strongly reflect the emphasis and terminology of the reform text. It 
presents topics "geometrically, numerically, and algebraically" (Hughes- 
Hallett et al., 1992, p. v). Only the reform students used these terms. Reform 
students call the numerical form tables. Instead of being concerned with 
graphing by hand, reform students stress with the behavior of the functions- 
where they are increasing or decreasing. One can discuss the slope of a 
tangent in terms of local linearity. The examples reform students use come 
from real life and promote understanding, input/output terminology in the 
reform classes replaces the domain/range language of the traditional classes. 
Even though the reform subjects had experienced a traditional approach to 
functions in their high school algebra classes, their distinctive terminology 
indicates that thfey are assimilating, at the very least, the vocabulary used in the 
reform calculus classes and textbook. It may also point to a difference in the 
content being covered in the two classes. 

A comparison of the experts’ core concepts completes this section. For 
th<=» restricted task, the combined experts' core list contains 25 terms (see Table 
1 ). Fifteen of the terms (60%) are on both the reform experts' core list and the 
traditional experts' core list. I looked at the concepts that were only on the 
reform experts' core list to determine any groupings that might distinguish them 
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from the concepts only on the traditional experts' core list. No strong differences 
emerged. The reform experts' core list had two terms, representations and 
table, that are emphasized more in reform classes. The traditional experts' core 
list contained max/min and increasing, concepts used primarily by the 
traditional students. Terms from the professors' different curricula do distinguish 
the core lists, but only minimally considering the size of the lists. 

On the experts' unrestricted concept mapping task, the eight professors 
generated 197 different concepts. The combined experts' core list contained 
only 10 concepts, showing the great diversity of these maps (see Table 1). All 
eight of the traditional experts' core concepts were on the combined experts' 
core list. The reform experts' core list had 14 concepts, only 7 of which are on 
the combined experts' core list. The other seven concepts relate more to higher 
mathematics than to the reform curriculum. 

In summary, qualitative analysis of the lists of concepts gleaned from the 
concept maps diu show distinctions between the two student groups, differences 
in terminology that linked each group to its respective curriculum. The 
qualitative analysis also reflected differences between the experts and the 
combined student group. The experts' combined core of concepts shows at 
least four higher-level categories-definition, class or type, properties, and 
operations~all of which are virtually non-existent in the students' maps. A 
qualitative study of the experts' core lists did not reveal any major differences 
along curricular lines. 

General Analysis of the Concept Maps 

Since this set of data did not lend itself to further quantitative analysis, the 
qualitative study of the concept maps took on added importance. Once again. I 
looked for differences between the two students groups that would correlate to 
the differences in their curricula or to their respective experts. A study of the 
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maps general appearance and structure did yield insights about the two student 
groups and about the experts. This sections details those findings. 

The most striking observation about the students' maps, and this holds for 
both reform and traditional groups, is that many of their concepts and 
propositions were trivial or irrelevant. For example, many of the maps showed 
an emphasis on variables, even listing x, y, and z as concepts. Another student 
termed them "letters" and has three concepts under letters; a-z, Greek, and x's 
and y's. Thirteen of the 14 students from the traditional group have some 
reference to variables on their maps, as do 12 of the 14 students from the 
reform classes. Two students from each group also connected slope to rise 
over run, a concept taught in first-year algebra and far from the topic of function. 
Often the students list concepts closely tied to the types of exercises they did in 
class, such as finding maxima and minima and classifying them as "absolute" 
("global" for the reform classes) or "local." One could say their preoccupation is 
with the "trees" and not the "forest." 

Some maps had sections that completely veered from the topic. One 
might hypothesize the students were simply trying to fill up the page or use up 
the time. Figure 8 shows half of a map on which there are no actual 
mathematical terms (the other half was little better). Another map contains the 
propositional chain "solutions help to get good grade may produce job lead $ 
pursue happiness." Figure 9 shows one student's complete map. Were it not 
for the concepts integrals and derivatives, one would not know this student had 
been in a calculus course. 

The second most noticeable characteristic of the students' maps was their 
algorithmic nature, particularly among the students from traditional classes. By 
algorithmic nature, I mean that instead of giving concepts and the relationships 
among them, the students gave steps in a procedure. For example, one student 
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has the chain function can be continuous if you can draw without lifting 
your pencil or they have no undefined points such as f(x)=1/x at x = 0 is a 
hole' at X = 0 which is incontinuous." She starts out with the concepts of 
function and continuous but then drifts into procedural steps. The algorithmic 
portions of the maps can usually be identified by the nature of the linking words, 
such as "by plotting" or "by locating," or by long phrases in the concept ovals, 
such as "it passes the vertical line test if f~\y) =x~\" 

Seven of the traditional students (50%) had groupings where algorithms 
or processes are evident, while only four (29%) from the reform classes did. The 
two most extensive maps from the traditional group show heavy algorithmic 
influence and could almost serve as procedural outlines of chapters in a 
traditional text. No map from the reform group showed extensive algorithmic 
groupings. 

The reform text regularly presented functions in different representations 
I studied the student maps to determine, if possible, a student's predominant 
view of function-whether it was an equation, a graph, a set of ordered pairs, or 
perhaps something else. The linking words played an important part in this 
analysis. For instance, one student's map has these propositions: "function 
consists of variables," "function can be graphed," 'function can be 
polynomial. " Using the same link for graphed and polynomial indicates the 
same connection: possibility, not necessity. These propositions, along with 
heavily algorithmic portions of her map, indicate an equation view of function 
Other maps I judged to have an equation perspective had graph separated from 
function by other propositions. Some students said straightforwardly, 'function 
{sic) are equations" 

Three students from the traditional group (21 %) and four from the reform 
group (29%) made a connection between function and real life situations, but 
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from the two groups one gets a different sense of the relationship. The three 
students from the traditional group had these propositional chains: "functions 
serve to represent complex problems, e.g. velocity," "function uses 
mathematical interpretation of real life situations," "equations discover 
natural phenomena links velocity and acceleration example balls falling in 
air." Velocity and acceleration are the typical examples in a traditional text. The 
least extensive example from the reform group contained the chain 'tunction is 
found in the real world like economics and engineering and medical field." 
Another reform student connects three chains and conveys the important 
concept of using functions to make predictions from collected data. A third 
student from the reform group shows that functions serve as approximations to 
real-life situations. A fourth map from the reform group has these propositional 
chains: "functions don't always involve equations some are about real life 
situations an example death rate of a population as a function of t^me, " and 
"functions involve modeling an example exponential decay--an example 
interest rate for a savings account." While the evidence is not conclusive, the 
concept maps indicated that students from the reform group had a better 
understanding that functions may be used to model actual, real-life situations. 

I also examined the maps to see if they reflected knowledge about the 
definition of function. In the group from the reform classes, one student has 
domain and range as concepts. Four others have input and output. None of 
the reform students indicated that each element of the domain can be paired 
with only a single range element, an essential part of the function definition. In 
the group from the traditional classes, four simply listed domain and range as 
concepts. Three others used domain and range and included the requirement 
about unique values for the range elements on their maps. 



Carol G. Williams 



Concept Maps as Research Tools in Mathematics 






Did the maps reveal any differences in hierarchy and integration of 
concepts between the two groups? In a word, no. Few maps showed any 
significant hierarchical structuring, although I did show them examples of 
hierarchical maps. The number of concepts emanating directly from function 
ranged from 1 to 1 3, with an average of 7 for both groups. The branches that 
did have several levels generally delineated procedures rather than linked 
coi loeijls. Integration of concepts, as shown by number of cross links, virtually 
did not exist. I found only two instances of cross links that showed an important 
connection, such as the inverse relationship of differentiation and integration. I 
had shown the students examples of cross links and had stressed their 
importance as part of the instruction on concept maps. While several students 
drew cross links, most were trivial, for example, 'Variables can be letters." 

Complete analysis of the students maps required comparison with the 
experts’ maps. Again, I have chose to use the experts' restricted maps, since 
they are much closer to the students' maps in content. Inasmuch as the experts 
are all PhD’s in mathematics, I assumed all their concepts and propositions to be 
valid and relevant. 

Unlike many of the students' maps, the experts' maps showed no hint of 
algorithms. Instead, they reflected many categorical groupings, several of which 
I discussed in detail in the section above on core lists. None of the experts 
demonstrates the students' propensity to think of a function as an equation. 
Instead, they define it as a correspondence, a map, a pairing, or a rule. All 
incorporate a definition in their map. Five experts (3 traditional, 2 reform) give 
real-world examples or allude to them. Looking at the overall content and 
complexity, the experts' maps as a group show much more homogeneity than the 
students'. 
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Distinguishing between the two expert groups' concept maps is difficult. 

As I have noted, ail experts referenced the definition and none gave algorithmic 
groupings, which points to homogeneity rather than divergence. In looking at 
the structure of the maps, one notes that two experts from each group drew 
hierarchical maps. One expert from each group drew a spider map, one expert 
from the traditional group used an unstructured web, and one expert from the 
reform group used a narrative style. (The expert who used the narrative style 
told me when I gave him his concept map instruction that he had done concept 
maps before in a curriculum development setting. His map more closely reflects 
that setting than the example maps I showed him.) I redrew all but the narrative 
map in a hierarchical style, realizing 1 might well be making inferences the 
experts never intended. The redrawn maps showed three to five levels of 
hierarchy with no distinctions between the two expert groups. They also showed 
that two of the reform experts' maps had a large number of cross links, 14 and 
17, while the three traditional experts' maps with cross links had a total of 1 1 
between them (5, 5, and 1). 

To summarize, a qualitative analy . i of the maps as a whole did indicate 
two differences between the reform and traditional student groups. The 
traditional group's maps displayed more and larger algorithmic portions. The . 
traditional studehts' maps virtually all pointed to an equation view of function, 
while a good number of the reform students allowed other viewpoints. The 
reform students may also have a different, broader view of function in real life 
The differences between the students' maps taken altogether and the experts' 
maps altogether were much more striking. The reform experts were unanimous 
in their lack of algorithmic portions while 39% of the student maps included 
algorithmic groupings. All the expert maps referenced the definition, but only 
43% of the students gave terms that could be loosely considered as referring to 
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the definition. The expert maps contained higher-level categories and were 
more homogeneous than the students maps, which contained trivial and widely 
varied groupings. Comparing the reform experts' maps to the traditional experts’ 
maps yielded no major differences. 

Implications for concept maps as research tools 

A major purpose of this study was to explore the use of the concept map 
as a research tool in the area of mathematics, particularly as it reflects 
conceptual understanding. The degree to which concept maps describe a 
person's actual mental representation is, of course, impossible to know. 
Nevertheless, the general homogeneity of the experts' maps and their distinct 
variance from the students' maps lend credibility to the conclusion that concept 
maps do capture a representative sample of one's conceptual knowledge. 

The concept maps in this study shed some light on the issue of hierarchy 
in knowledge representation. As mentioned in the literature review, Novak and 
Gowin (1984) require hierarchical maps, since they base their maps on 
Ausubel's (1968) hierarchical view of knowledge. Other researchers (Hiebert & 
Lefevre, 1986; White & Gunstone, 1992) allow for hierarchy. For White and 
Gunstone, certain knowledge domains are hierarchical while others are not. In 
this study I gave no hierarchical restraint. Consequently, four of the eight 
experts did not create a hierarchical map but rather made a web or spider map. 
Since the experts did not all draw hierarchical maps, it appears function is not 
inevitably a hierarchical domain. Yes, one can redraw the web maps in a 
hierarchical fashion, but such a process necessarily requires the researcher to 
make her own assumptions about the relative levels of hierarchy. These 
second-party assumptions may not be consistent with intentions or perceptions 
of the original map maker. 
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Some might argue that the diversity in the experts' maps resulted from 
inadequate instruction on how to draw concept maps. Rogers' (in press) study 
speaks to this point. Despite detailed instruction on hierarchical maps to 
students who had drawn 20 concept maps prior to the study, Rogers reports the 
students' maps showed three patterns of conceptual organization; hierarchical, 
spider (she terms them "branching"), and random. It appears the students' 
personal organization of knowledge took precedence over the researcher's 
prolonged, explicit instruction. I would argue the subjects in my study received 
adequate instruction for their level of expertise and that differences in the maps’ 
organization reflect their personal views. 

Before concept maps become an accepted research tool in mathematics, 
researchers must resolve issues concerning numerical scoring schemes. 

Judging from the data I collected and from other studies that have used concept 
maps, the hierarchical constraint provides one important key to creating a valid 
scoring scheme for concept maps. Another key is whether one gives the 
subjects several concepts to use in the maps or merely supplies one main topic. 
Stuart (1985) describes this process cf creating a map from a single given topic 
as "constructing maps de novc " The six studies (Beyerbach, 1986; Coleman, 
1993: Laturno, 1994; Markham et al., 1994; Park, 1993; & Wallace & Mintzes 
(1990)) most relevant to this study all worked in the Novak and Gowin tradition, 
required hierarchical maps, and gave the maps a numerical score. Beyerbach 
(1986) is the only one who used concept mapping de novo, and her scoring 
scheme is different from most research in the Nov;ik and Gowin tradition. I did 
not find her scoring scheme to be valid for measuring the degree of knowledge 
differentiation. (For more details, see Williams, in press.) 

None of the studies convinced me that valid and reliable scoring schemes 
are currently available for concept maps constructed de novo. It does appear 
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that training students on hierarchical maps, giving them core concepts with 
which to build their maps, and limiting the subject domain can create a setting in 
which numerical schemes are valid. Yet in my view, each of these practices 
reduces the usefulness of concept mapping as a representation of a subject's 
own personal knowledge structure. 

As with all research tools, the concept map has limitations. Concept 
maps created de novo display so much diversity that they are difficult to score 
numerically, particularly if they do not exhibit a hierarchical structure. White and 
Gunstone (1992) liken scoring non-hierarchical maps to scoring an essay-one 
looks for the overall point of view rather than specific knowledge content. If a 
researcher's aim is primarily to gather quantitative evidence to support an 
hypothesis, a more structured concept map task can be appropriate. For 
example, a task where one gives subjects terms to interrelate is better if the 
researcher is particularly interested in how subjects see the connections 
between those particular terms. However, if one is looking for an individual's 
knowledge structure in its "purest" most personal form, the unstructured mapping 
task is better because it does not put concepts into the subjects' heads. In this 
study, where concept maps on function were used for the first time, the 
unstructured task proved appropriate and provided useful information about the 
broad topic. 

The results of this study suggest several areas relating to concept maps 
where further research is needed. One is the area of knowledge hierarchy. A 
second is numerical scoring schemes Would training the experts and students 
to draw hierarchical maps substantially alter the findings? The researcher could 
teach the experts to perform the restricted task (relating to first year calculus 
concepts) by drawing hierarchical maps. One could then compare those 
hierarchical maps to the ones drawn for this study without the hierarchical 
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stipulation. A variation on t[;ie same study would give the experts the 25 
concepts on the combined experts' core list and ask them to draw a concept map 
limited to these concepts. This could be done with or without the hierarchical 
constraint. One could study how the maps differed from the de novo expert 
maps drawn for this study, determining if the experts organized their concepts 
differently or if they added or deleted categories. One could ascertain whether 
these maps lend themselves to a scoring scheme. And would experts draw 
hierarchical maps as instructed or would they choose to ignore that instruction 
as the subjects in Rogers' (in press) study did? 

Stuart (1985) states the need for a more holistic and qualitative scoring 
technique in order to make maps created de novo more useful in research. I 
think concept maps created de novo can be useful in research without a 
numerical scoring technique. While numerical scoring helps compare different 
groups, de novo concept maps as drawn also show major differences between 
groups. Numerical scoring may conceal the idiosyncratic nature of the maps. 
Often, researchers find new ideas and relationships in these idiosyncratic maps. 
As the mathematics community continues to promote meaningful learning and to 
look for conceptual links, the concept map places an appropriate tool at the 
researcher's disposal. 
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